Combined Experience Replay (CER)

1 Overview

S. Zhang and R. Sutton discussed the effect of replay buffer size1. After the famous DQN paper2, almost all experiments fixed the buffer size to \(10^6\). Their experiments indicated that larger buffer made learning slower. Combined Experience Replay (CER) is a method where a latest transition is mixed with transitions from replay buffer. CER improves learning speed, especially for large replay buffer.

2 With cpprb

You can sample (batch_size - 1) transitions from replay buffer and add the latest transition to the batch.

  1. S. Zhang and R. Sutton, “A Deeper Look at Experience Replay”, NIPS (2017), (arXiv cs.LG 1712.01275) ↩︎

  2. V. Mnhi et al., “Human-level control through deep reinforcement learning”, Nature 518, 529-533 (2015) ↩︎